Using if ($(current).closest('li', this.core.editor()).length !== 0) instead is enough as it limits the search inside Redactor.

Your proposed change first searches the closest list item and then checks if it is contained within Redactor (hint: there is also this.utils.isRedactorParent() for this), which is essentially the same as above but in reverse.

Ah, your way is more efficient indeed!
if ($(current).closest('li', this.core.editor()).length !== 0) does not work though, because this.core.editor() returns a set of elements.

I've changed it into if ($(current).closest('li', this.core.editor().get(0)).length !== 0) which does work.

Let me know what you think!

My bad, it's been a while I used closest(), just did a quick check of the API docs and didn't notice it required a DOM element rather than a jQuery object; Your solution is correct.

No worries ;-)